Centralized Management of Distributed Data Sources

ABSTRACT

Aspects of the disclosure are directed to a central management plane (CMP) of one or more processors for regulating streams of data from each of a number of network nodes of a distributed network. The one or more processors can train and deploy machine learning models across the network nodes, and the CMP can generate policies for each network node. The generated policies specify how a network node is to transmit data to the platform for further training or retraining of the deployed machine learning models. The CMP generates the policies using metric data characterizing each network node and respective streams of input data, and are generated based on a number of objectives, including model output quality of the deployed models, and operational cost to transmit and process streams of data across the distributed network.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the filing date of U.S.Provisional Pat. Application No. 63/243,252 filed Sep. 13, 2021, thedisclosure of which is hereby incorporated herein by reference.

BACKGROUND

Computing platforms can provide a number of services across adistributed network of computing devices. Computing devices sendingrequests to a computing platform are often physically remote fromservers of the platform implementing an offered service. These computingdevices communicate directly with the computing platform, or indirectlywith the computing platform through one or more other devices orcommunication networks. Services offered by a computing platform can bedeployed remotely on computing devices to handle requests remote fromthe computing platform. Machine learning models can be deployed toremote devices and trained or retrained on a central platform connectedto the remote devices.

BRIEF SUMMARY

Aspects of the disclosure are directed to a system for improving thedeployment of machine learning models across nodes of a distributednetwork, by regulating the flow of data to the system from each networknode used to train models deployed on each network node. A datamanagement system generates per-data stream policies for each datastream from each network node of a distributed network transmitted to acentral platform for training machine learning models. The policyspecifies how each network node is to regulate streams of input datafrom the node and transmit the data to a central platform implementingthe data management system. Regulation of a data stream can refer toaltering characteristics of the data stream, such as the type, volume,and rate of data passing from the node to the central platform. Bydefault, data streams can be transmitted over a distributed networkaccording to predetermined parameters specifying the rate, type, and/orvolume of data transmitted in the streams.

Policies are generated in accordance with a plurality of objectives,such as the output quality of a model deployed on a network node, andthe operational cost, for example, measured in network traffic bandwidthor processing cycles, to transmit data from the network node to the datamanagement system. The data management system, through policiesgenerated for each data stream, directs the quantity and quality of datato be transmitted from each node to the platform, reducing networktraffic while providing enough data that the central platform cancontinue to train or retrain deployed models to maintain minimumaccuracy or performance benchmarks.

An aspect of the disclosure includes a system, including: a distributednetwork of a plurality of network nodes; and one or more processors inthe distributed network, the one or more processors implementing acentral management plane (CMP), the CMP configured to: receive metricdata at least partially characterizing a first network node of theplurality of network nodes executing a deployed machine learning modelon a stream of input data received by the network node from a datasource; generate a policy for regulating the stream of input datatransmitted by the network node to the one or more processors; and sendthe policy to the network node.

An aspect of the disclosure includes a computer-implemented method,including: receiving, by one or more processors, metric data at leastpartially characterizing a first network node of a plurality of networknodes executing a deployed machine learning model on a stream of inputdata received by the network node from a data source, the plurality ofnetwork nodes part of a distributed network; generating, by the one ormore processors, a policy for regulating the stream of input datatransmitted by the network node to the one or more processors; andsending, by the one or more processors, the policy to the first networknode.

One or more non-transitory computer-readable storage media encodinginstructions that when executed by one or more processors, cause the oneor more processors to perform operations including: receiving metricdata at least partially characterizing a first network node of aplurality of network nodes executing a deployed machine learning modelon a stream of input data received by the network node from a datasource, the plurality of network nodes part of a distributed network;generating a policy for regulating the stream of input data transmittedby the network node to the one or more processors; and sending thepolicy to the first network node.

The foregoing and other aspects of the disclosure can include one ormore of the following features, alone or in combination. For example,some aspects include all of the following features, together incombination.

The policy can include one or more rules for adjusting one or more of:the rate of the stream of input data transmitted by the first networknode to the CMP, the volume of the stream of input data transmitted bythe first network node, a time at which the stream of input data istransmitted by the first network node, and the types of data in thestream of input data transmitted by the first network node to the CMP orthe one or more processors.

The CMP or one or more processors can be further configured to receive aregulated stream of input data, wherein the regulated stream of inputdata is the stream of input data after the policy is applied to thestream of input data; and wherein in generating the policy, the CMP orthe one or more processors are further configured to generate the policyaccording to a plurality of objectives based at least on: reducing theoperational cost of transmitting the regulated stream of input data overthe distributed network or using the regulated stream of input data totrain the deployed machine learning model, and increasing the outputquality of the deployed machine learning model according to one or morequality metrics, after training the model on the regulated stream ofinput data.

The operational cost of transporting output data can be further based onone or more of: the rate at which the regulated stream of input data isreceived by the CMP over a period of time, the volume of the regulatedstream of input data received by the CMP, and the type of data in theregulated stream of input data.

The one or more quality metrics of the machine learning model canmeasure one or more of the inference accuracy, the inference precision,and the inference recall of the machine learning model after training orretraining the machine learning model on training data including theregulated stream of input data.

The machine learning model can be a first machine learning model, andwherein in generating the policy, the CMP or one or more processors canbe configured to process the metric data through a second machinelearning model trained to generate policies, the second machine learningmodel trained using labeled metric data: including featurescorresponding to one or more of the inference accuracy, the inferenceprecision, and the inference recall of the deployed machine learningmodel, and/or labeled with data characterizing one or more of the rateof the stream of input data, the volume of the stream of input data, andthe types of data in the stream of input data transmitted by the networknode to the CMP or one or more processors.

The plurality of objectives can further include one or more objectivesfor training the machine learning model.

The stream of input data can be a first stream of input data, the policyis a first policy, and the metric data is first metric data; and the CMPor one or more processors can be further configured to: receive secondmetric data at least partially characterizing the performance of thefirst network node executing the deployed machine learning model on asecond stream of input data different than the first stream of inputdata, and generate and send a second policy for regulating the secondstream of input data, the second policy different from the first policy.

The second stream and the first stream of input data can be receivedfrom the network node by the same data source.

The CMP can be further configured to: receive an indication of a secondnetwork node added to the distributed network; receive metric data fromthe second network node; determine, based on the metric data of thefirst network node and the metric data of the second network node, thatthe first and second network nodes are similar within a predeterminedthreshold; and in response to the determination, send the policy to thesecond network node.

The metric data can be first metric data, the policy is a first policy,and the CMP can be further configured to: receive second metric datafrom a second network node of the plurality of network nodes; generate asecond policy using the first metric data, the second metric data, orboth the first metric data and the second metric data, wherein thesecond policy is different than the first policy; and send the secondpolicy to the second network node.

In regulating the stream of input data, the CMP or the one or moreprocessors can be further configured to apply one or more rules of apolicy to the stream of input data to adjust the transmission of data inthe stream of input data.

The metric data at least partially characterizing a first network nodeof the plurality of network nodes can include one or more of: physicalcharacteristics of the distributed network or of the first network node,characteristics of one or more streams of data received by the networknode, and characteristics of the data source.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example data management system incommunication with network nodes, according to aspects of thedisclosure.

FIG. 2 is a block diagram of the example data management systeminteracting with a network node, according to aspects of the disclosure.

FIG. 3 is a flowchart of an example process for regulating a stream ofoutput data for a network node of a distributed network.

FIG. 4 is a flowchart of an example process for initializing a newnetwork node of the distributed network, according to aspects of thedisclosure.

FIG. 5 is a flowchart of an example process for regulating a stream ofoutput data on a network node of a distributed network, according toaspects of the disclosure.

FIG. 6 is a block diagram of an example computing environment forimplementing the data management system.

DETAILED DESCRIPTION Overview

Aspects of the disclosure are directed to a system for improving thedeployment of machine learning models across nodes of a distributednetwork, by regulating the flow of data used as training data to thesystem from one or more data sources in communication with each networknode. A data management system generates policies for each data streamof a network node of a distributed network. The policy specifies howeach network node is to regulate each stream of input data from the nodeto a central platform implementing the data management system.Regulating a data stream can refer to altering characteristics, such asthe type, volume, and rate, of data passed through a distributed networkto the central platform. Before regulation, the data stream can becharacterized at least by a rate, volume, or type of data present in thedata stream. Initially, the data stream may be subject to predeterminedparameters, for example defined by a network node, by the centralplatform, and/or by another device configured to control the distributednetwork, for its transmission over the distributed network. Forinstance, a network node may be initially configured to transmit as muchdata as possible, limited by network bandwidth and/or the node’sprocessing capacity to transmit data.

Data can be provided to a computing platform for training one or moremachine learning models on each of a variety of different network nodes.One problem with transmitting as much data as possible to the computingplatform is that incremental increases in the amount of data providedmay not result in corresponding improvements to a model trainedaccording to the additional data.

The data management system according to aspects of the disclosuregenerates a policy in accordance with a plurality of objectives, such asimproving the output quality of a model deployed on a network node, andlowering the operational cost, for example measured in network trafficbandwidth or processing cycles, to transmit and process the stream ofdata from the network node to the data management system.

Network nodes can be physically separated over a large distance andacross one or more interconnected networks, compounding the cost totransmit and process data streams of input data. The input data can beused to generate new training data, which can be used by a centralcomputing platform for training or retraining a machine learning modeldeployed on a network node. As training a machine learning model is acomputationally expensive task requiring large amounts of time andcomputational resources to process training data, a computing platformcan train models to be deployed or redeployed on a network node.

The data management system balances model output quality withoperational cost to transmit additional data from the node to the systemfor training or retraining the deployed machine learning model. In otherwords, the data management system can provide just enough data to meetpredetermined output quality thresholds, for example defined to providea minimum level of quality of user interaction with a node implementinga deployed model. The system can be trained to identify characteristicsof a data stream, such as a type, volume, and rate of data, which, whenused for training a machine learning model, causes the system togenerate a model with an accuracy at least meeting or exceedingpredetermined output quality thresholds. The system balances relativeperformance gains from added training time to a model, with theoperational cost for transmitting and processing the additional datafrom the network node to the platform. As an example, additional inputdata used to train the model may realize a narrow, but quantifiable,increase in performance in the retrained model. However, the additionalinput data may place a strain on a distributed network and inhibit theperformance of the deployed model in other ways, for example measured innetwork latency, response time, etc.

The system can provide a policy to the network node to regulate, forexample, the type, volume, and rate at which data is transmitted to thesystem, balance objectives, such as operational cost to transmitadditional data to the platform, with model quality improvements as aresult of training on the additional data. For example, the policy maydefine certain time periods at which data transmitted is not to exceed apredetermined threshold, as a way to regulate the rate at which data istransmitted to the system from the network node. Other objectives caninclude objectives related to training the deployed model to perform aparticular task. For example, additional objectives can include reducingor mitigating bias in input data received by the system used in trainingor retraining the deployed model. Bias can be quantified by one or morestatistical measures. Data sources in communication with the networknode can be, for example, individual user computing devices, such asmobile phones or personal laptops; one or more or more servers; or anyof a variety of computing devices, including wearable devices and othersensors, embedded systems, or other devices configured to communicatewith the network node. The way in which the data source communicateswith the network node can vary, for example, over a radio accessnetwork, a core network, or as part of an operational support system. Aspart of generating the policy, the data management system is configuredto receive characteristics of the network node, which can include thetype or types of networks connecting the system to the node. A datamanagement system as described herein can identify patterns between datastreams having certain characteristics, with higher levels of modelperformance as a result of training models on those data streams. Usingthese patterns, which for example can be learned using a machinelearning model, the system can generate the policies specifying how thenetwork node is to regulate data streams to the system, for example byadjusting the rate, volume, and/or type of data transmitted to theplatform implementing the system.

As an example, the data management system can be deployed on a computingplatform in communication with various network nodes of atelecommunications network. Different nodes may have different modelsdeployed on each node, for example for analyzing streams oftelecommunication network data passing through each node, or forautomating some processing task that receives telecommunication networkdata as input. The deployed models may be subject to minimum outputquality thresholds, for example a minimum recall rate or a maximumtolerated false positive rate, such as when a deployed model is trainedto generate an output classification based on ingested telecommunicationnetwork data. As a telecommunication network can be spread far alongmany physical or virtual computing devices, the various models can bedeployed on devices physically proximate to data sources oftelecommunication network data. At least some data of each data streamis transmitted by a network node to the data management system, as oneor more regulated streams of data.

The data management system can be implemented on one or more computingdevices, for example computing devices of a computing platform. Thecomputing platform may be connected to the telecommunications networkover a separate connection, and/or be part of the telecommunicationsnetwork itself. The regulated streams of data are ingested by the datamanagement system and used to train or retrain, for example to updateweights of, machine learning models deployed on the network nodes.

After a period of time, for example a predetermined period of timeand/or in response to a request from a network node, the data managementsystem can provide an updated model to the network node. The datamanagement system can generate policies for regulating data streamsacross the various telecommunication network nodes. The network nodescan be configured to regulate data transmitted to the data managementsystem, according to a received policy. Data can be transmitted moreefficiently, for example less data or during less network-congestedperiods of time, without substantially reducing the model performance ofmodels trained on the regulated data by the data management system. Inthe example of a telecommunications network that may have many differentnodes receiving data from a variety of different smaller andheterogeneous networks, implementing the data management system asdescribed herein can reduce the burden of the network in transmittingdata.

Each network node in communication with a data management systemaccording to aspects of the disclosure can include one or more nodemetric engines and one or more data source regulators implemented onnodes of the distributed network. The data management system can includea central management plane (CMP). The CMP receives node metrics datafrom the respective metric engine implemented on each of multiple nodesof a distributed network. The CMP uses the received node metrics data togenerate a corresponding policy of actions to perform, or conditions toenforce, for each node of the distributed network. In some examples, theCMP implements a machine learning model trained to generate policies foreach node.

The CMP can be trained with labeled metric data, which can includefeatures of a deployed machine learning model characterizing one or moreof the inference accuracy, the inference precision, and the inferencerecall of the deployed machine learning model. The metric data can belabeled for example, with data characterizing one or more of the rate ofthe stream of input data, the volume of the stream of input data, andthe types of the data in the stream of input data received by the CMPfrom the network node.

The corresponding data source regulator for a network node receives apolicy from the CMP, and performs actions defined by the policy toregulate the stream of input data from the node. For example, the policycan specify a maximum rate, for example, in bits per second, at whichthe stream of input data is to be provided to the CMP.

Because nodes of the network can be heterogeneous, for example, are indifferent geographic locations, with different supportinginfrastructure, sources of data, data traffic patterns for data to andfrom each node, etc., the system generates different policies for eachdata stream received from each node. The data management system accountsfor deployment-specific characteristics of each node, as well asspecific characteristics of different data streams from differentsources of data. The system can generate different policies, even whenthe base model or task performed at each node is the same for eachreceived stream of data. In this way, the system can receive less datafor training one or more different models deployed on a respective node,using a policy that specifies regulated characteristics of each of oneor more data streams, which may be received by the node from one or moredata sources. Data stream characteristics can vary, and training datacan affect model performance in different ways. Therefore, providingmultiple policies for multiple data streams can reduce or eliminateadverse effects, such as reduced model quality, as compared withapproaches in which a single policy is predetermined and applied to alldata streams to the system.

The data management system allows for rapid scaling of new deploymentsof nodes in the distributed network, and can provide an initial policybased on similarly-deployed nodes to accelerate integration of the nodeinto a distributed network. New nodes can be deployed faster at leastbecause the initial policy can be provided versus deploying the nodewithout a policy, and/or versus deploying the node with a uniformpredetermined policy which may not be suited given particularcharacteristics of the network node and/or data sources, or datatransmitted to the node. The data management system can update theinitial policy upon receiving metric data, for example, related togeographic location of the node, output quality of the model currentlydeployed on the node, traffic patterns of the node includingcharacteristics about individual data streams, etc., characterizing thedeployment of the new nodes. Faster deployment of nodes can improve thesystem’s capability to add additional computing resources whennecessary, and can reduce idle time between receiving a request foradditional nodes of resources in the network, and the initialization ofthe requested node.

Aspects of the disclosure provide for a number of technical advantages.Machine learning models can be trained at a central platform anddistributed at different physically remote nodes, for example at cell,near, and far edges. A platform implementing a data management system asdescribed herein can adjust the deployment of each model acrossheterogeneous network nodes, including adjusting the rate at which datafrom the node is transmitted to the platform. The policy reflects avariety of different factors unique to each node and data streams fromdata sources in communication with the node. The operational cost, forexample measured in network bandwidth and/or in processing cycles, andperformance, e.g., model accuracy, of training the node-deployed modelsat a platform can be balanced to reduce operational cost in transmittingdata over a distributed network, without substantially reducing modelquality and performance. The distributed network can use additionalresources saved as a result of sending regulated data to the system forother purposes, for example in deploying additional models to thenetwork node and increasing its capability to serve user requests. Thedata management system, through policies generated at a per-data streamlevel, allows for just enough data to be transmitted from each node tothe platform, reducing network traffic while maintaining minimumaccuracy or performance benchmarks for each node.

Generating per-data stream policies allows for more granular adjustmentto data streams transmitted to the system, in turn allowing for moreefficient data transmission to the system for training a model deployedon a network node, even when more efficient adjustments to regulatingone data stream may adversely affect another data stream of the samenetwork node. Each data stream can be regulated, e.g., characteristicsof the data stream can be adjusted, individually, at least by thegeneration and execution of per-data stream policies as describedherein.

As the network scales in size, for example as additional network nodesare added, the system can initialize newly added nodes with policies ofother nodes identified as similar to the newly added nodes within athreshold. In doing so, the system allows for quickly improving theperformance of the newly added node at reduced operational cost for datastream transmission, before later fine-tuning the policy to reflectcharacteristics specific to the node and of data streams received fromthe node. This added bootstrapping of a previously-generated policy canreduce the time to deploy a new network node, which not only directlyimpacts the capability of the platform in receiving and servingprocessing requests, but allows the platform to react faster to addingnew computing resources when the need is identified.

Example Systems

FIG. 1 is a block diagram of an example data management system 100 incommunication with network nodes 110A-B, according to aspects of thedisclosure. The system 100 can be implemented, for example, as part of acomputing platform, communicating with the network nodes 110A-B over anetwork 120. As described in more detail with reference to FIG. 6 , thecomputing platform can include a number of computing devices, such asserver computing devices, which can communicate with a number of otherdevices, such as devices implementing the network nodes 110A-B.

The network nodes 110A-B can be part of a number of network nodesconnected over the distributed network 120. Examples of network nodesinclude user computing devices, such as personal computing devices,wearable devices, or smartphones. In some examples, network nodes caninclude one or more computing devices in communication with a network ofother computing devices. The network node can implement an inferencingengine that is configured to receive input data and requests to processthe input data, from the network of computing devices. Network nodes canbe implemented in a variety of different locations, for example acrossdifferent geographic regions. Network nodes can service a variety ofdifferent devices, for example corresponding to different users who mayor may not be affiliated with one another. Example locations in whichnetwork nodes may be deployed range from individual buildings to entirecities, and locations of at scales in-between.

The network nodes 110A-B can receive data from a number of data sources115A-E. As with the network nodes 110A-B, the data sources 115A-E caninclude any of a variety of different computing devices, includingcomputing devices serving as a proxy between a network node and one ormore other devices, for example devices in a local network. The datasources 115A-E in communication with network nodes 1120A-B can be, forexample, individual user computing devices, such as mobile phones orpersonal computers; one or more servers; or any of a variety ofcomputing devices, including wearable devices and sensor devices,embedded systems, or other devices configured to communicate with thenetwork node. The way in which the data source communicates with thenetwork node can vary, such as over a radio access network, a corenetwork, or as part of an operational support system.

Each data source sends a respective input data stream 116A-E to thenetwork nodes 110A-B. Each input data stream includes data that is inputto the inferencing engine of the receiving network node. In response,the network node can process input data in the received stream andgenerate output data in response. The output data can be generated bythe inferencing engine, for example by processing the input data througha machine learning model trained to process the input data. Some datasources, such as data source 115A, can send multiple input streams to anetwork node, such as input streams 116A, 117A to the network node 110A.The separate streams can correspond to input received from a largernetwork of devices behind the data source 115A. For example, the datasource 115A can be one or more computing devices, with the network nodeacting as a proxy between the network node 110A and one or more othercomputing devices or networks of devices. Each device or network ofdevices can send a respective stream of input data to the network 110A,either directly or through one or more proxy devices.

As described in more detail herein, the data management system 100 isconfigured to train machine learning models for deployment on thenetwork nodes 110A-B. The system 100 receives a stream of input dataregulated according to a policy generated by the system 100, and usesthe regulated stream as training data for training one or more machinelearning models. The task the machine learning models deployed as partof the inferencing engines 114A-B can vary depending on specificrequirements the network nodes 110A-B are configured to meet. Examplesof machine learning tasks which deployed machine learning models can betrained to perform follow.

As an example, the input to an inferencing engine of a network node canbe in the form of images or videos. The inferencing engine can beconfigured to extract, identify, and generate features as part ofprocessing a given input through one or more deployed machine learningmodels, for example as part of a computer vision task. Machine learningmodels trained to perform this type of machine learning task can betrained to generate an output classification from a set of differentpotential classifications. In addition or alternatively, the machinelearning model can be trained to output a score corresponding to anestimated probability that an identified subject in the image or videobelongs to a certain class.

For instance, the network node 110A can be part of a system formonitoring an industrial manufacturing process, in which objects aredesigned and/or manufactured. The data sources 115A-E can include one ormore sensors collecting sensor data at various points in a manufacturingline, including image or video data. The inferencing engine 114A canprocess the input data through a machine learning model trained todetect anomalies in manufactured objects, and flag those anomalies forfurther inspection and/or to take some predetermined action in responseto the detection.

As another example, the input to an inferencing engine of a network nodecan include data files corresponding to a particular format, such asHTML files, word processing documents, or formatted metadata obtainedfrom other types of data, such as metadata for image files. Machinelearning model(s) deployed as part of the inferencing engine can betrained to classify, score, or otherwise predict some characteristicabout the received input. For example, the machine learning model(s) canbe trained to predict the probability that the received input includestext relating to a particular subject. Also as part of performing aparticular task, the machine learning model can be trained to generatetext predictions, for example as part of a tool for auto-completion oftext in a document as the document is being composed. A machine learningmodel can also be trained for predicting a translation of text in aninput document to a target language, for example as a message is beingcomposed.

In the above example, data sources providing data in this example caninclude user computing devices, which can provide queries to the networknode including data files or plain text for processing. The usercomputing devices can interact with the network node over an interface,such as a web interface accessed through a web browser or applicationinstalled on the user computing device.

As another example, the input to the inferencing engine of a networknode can be audio input, including streamed audio, pre-recorded audio,and audio as part of a video or other source or media. Machine learningmodel(s) deployed as part of a network node inferencing engine can betrained to perform speech recognition, including isolating speech fromother identified sources of audio and/or enhancing characteristics ofidentified speech to be easier to hear. A machine learning model can betrained to predict an accurate translation of input speech to a targetlanguage, for example in real-time as part of a translation tool. Datasources can include user computing devices, such as wearable devices,including earbuds, headsets, etc., configured to communicate audio datain real-time for processing by a network node.

Other types of input documents can be data relating to characteristicsof a network of interconnected devices. These input documents caninclude activity logs, as well as records concerning access privilegesfor different computing devices to access different sources ofpotentially sensitive data. Deployed machine learning model(s) can betrained by the training engine 104 for processing these and other typesof documents for predicting on-going and future security breaches to thenetwork. For example, the machine learning model(s) can be trained topredict intrusion into the network by a malicious actor. As anotherexample, a machine learning model can be trained to classify anomalousdata from a set of input documents, and flag instances of predictedanomaly for further manual review and/or automatic correction. Datasources in this example can include computing devices in a localnetwork, configured to monitor and record network activity and forwardthe records to a network node deploying one or more machine learningmodels for processing the records, as described in this example andothers.

In addition to data input, including the various types of data describedherein, the inferencing engines 114A-B can be configured to preprocessfeatures corresponding to given input. Features are values, for example,numerical or categorical, which relate to some characteristic of theinput. For example, in the context of an image, a feature of the imagecan relate to the RGB value for each pixel in the image. The inferencingengines 114A-B can be configured to extract and select relevant featuresfor processing to generate an output for a given input, and can also betrained to generate new features based on patterns identified by thedeployed models between various characteristics of input data. In someexamples, the deployed machine learning model(s) of an inferencingengine can be trained to perform some or all of the featureprocessing/extraction for given input data.

FIG. 2 is a block diagram of the example data management system 100interacting with a network node 210, according to aspects of thedisclosure. FIG. 2 shows a data source 215 transmitting input datastream A 216A and input data stream B 216B to a network node 210. Thedata streams 216A-B are received by a data source regulator 214 and aninferencing engine 212 of the network node 210. The inferencing engine212 can include a node metrics engine 213 and a model 217. The model 217can be one or more of any of a variety of machine learning modelstrained to perform a machine learning task by processing an input datastream, as described herein with reference to FIG. 1 . The model 217 cangenerate output which can at least partially form a node output 232 thatis sent to the data source 215. The node output 232 can be a response tothe input data received from the data source 215. For example, the datasource 215 can pass a query or request to process some input data aspart of an input data stream. In other examples, the node output 232 canbe passed to other network nodes sharing a connection with the networknode 210 (not shown).

The inferencing engine 212 can receive the input data and request,process the input data according to the request, for example, accordingto any parameters for processing the input data provided as part of therequest, and generate a model output in response to the processed inputdata. The network 210 can send the model output and optionally anyadditional information to the data source 215. The data source 215 canreceive the node output 232, and send the output 232 to one or moreconnected computing devices, for example for continued downstreamprocessing. In some examples, instead of receiving a request to processdata from a data stream, the network node 210 is configured toautomatically process received data, for example as received oraccording to any of a variety of predetermined parameters.

The data source regulator 214 passes regulated input data streams A, B222A-B to the system 100. A regulated input data stream is an input datastream received from a data source after a respective data stream policyis applied to the input data stream, for example by the data sourceregulator 214. For example, the data source regulator 214 receives datastream policies A, B 226A-B from a central management plane (CMP) 228 ofthe system 100. A data stream policy can include one or more rules,which when applied by the data source regulator 214 to a data stream,adjusts the transmission of the data stream to the data managementsystem 100 in one or more ways. For example, the rules can specify therate of the stream of input data transmitted by the network node; thevolume of the stream of input data transmitted by the network node;and/or the types of data in the stream of input data transmitted by thenetwork node. The data source regulator 214 is configured to convert areceived policy into one or more instructions executable by the datasource regulator 214 to cause the data source regulator 214 to apply thepolicy in transmitting a data stream.

The CMP 228 generates the policy to meet a number of objectives, such asobjectives for reducing the operational cost of transmitting theregulated stream of input data over a distributed network or using theregulated stream of input data to train the machine learning model 217,and increasing the output quality of the deployed machine learning modelaccording to one or more quality metrics, after training the model onthe regulated stream of input data. The CMP 228 balances at least twoobjectives in generating a data stream policy, lowering the operationalcost to transmit and process input data for training as low as possible,while raising the output quality of models deployed on the network nodeand trained by the system 100 to meet predetermined output qualitythresholds. In other examples, the CMP 228 is configured to raise outputquality of models deployed on the network node and trained by the system100 as high as possible, while also balancing the operational costs totransmit streams of data for training.

In some examples, the CMP 228 may generate the policy according to otherobjectives, in addition to increasing model output quality and reducingoperational cost for transmitting and processing streams of input data.For example, the CMP 228 may generate the policy to reduce bias in thetrained machine learning model deployed on the network node. Bias is thedifference between outputs generated by a machine learning model and aground-truth or correct output, for a given input. The machine learningmodel may become biased for a variety of reasons, which can stem fromthe training data used to train the model. In addition to balancingoperational cost and model output quality, the CMP 228 can be configuredto generate policies to regulate data streams received by the datamanagement system 100 to reduce bias in a machine learning model trainedusing the regulated data streams. For example, the generated policy canspecify one or more filters that when executed by a network node, causesthe network node to filter out certain types of data that have beenidentified by the data management system 100 as biasing a deployedmodel.

The operational cost for transmitting the regulated stream of data, suchas the data streams 222A, 222B, can be measured in processing cycles orin computing resources for transmitting the data between the node 210and the system 100. The operational cost can be reflected, for example,in processing cycles required by either the node 210 or the system 100in sending and receiving the data streams, respectively; networkbandwidth required to transmit the data; time spent transmitting thedata and the latency between sending and receiving the data; and anylatency caused in other transactions across the network as a result oftransmitting the data stream, for example because other data was queuedand delayed while waiting for the data stream to be sent across thenetwork.

The operational cost of transporting input data can be based on thecost, for example in time or in number of processing cycles, fortransmitting a stream of input data to the system 100 at different ratesor volumes. The rate at which a stream is transmitted can be measured asunits of data over a period of time, such as megabytes per second.Higher rates generally incur a higher operational cost than lower ratesof data transmission. The volume at which a stream is transmitted can bemeasured as units of data, for example in gigabytes or terabytes. Highervolumes of data transmitted by the network node 210 to the system 100generally require more computing resources—and therefore have a higheroperational cost— to process, over lower volumes of data.

The operational cost for transmitting a data stream can also be based onwhen the data stream is sent to the system 100. For example, someperiods of time may correspond with less network activity, making theoperational cost to transmit the data stream lower, at least because thechance of delay in transmitting the stream, or other data as a result oftransmitting the stream, is lower. On the other hand, transmitting thedata stream during other periods of time may conflict with other datatransmitted during a period of peak network activity.

The operational cost for transmitting a data stream can also be based onthe type of data that is being transmitted or processed in the datastream. For example, some types of data, such as tensors or higher-orderdata structures, are generally more computationally intensive totransmit and process over other types of data, such as bit indicators ordata transmitted as un-encoded raw bytes. A data stream may include oneor more types of data, and the network node may transmit some, all, ornone of a type of data based on a received policy from the CMP 228.

The operational cost for processing the regulated stream of data canrefer to one or more measures of computing resources used in training amachine learning model using the regulated stream of data as input. Forexample, the operational cost can be measured in number of processingcycles and/or time in preparing the stream of data for training, andtraining, validating, and testing the model according to the preparedtraining data.

The CMP 228 balances lower operational cost with improving the outputquality of a model trained by the system 100 and deployed on a networknode. Output quality can be measured in a variety of different ways, forexample the output quality can be measured according to one or more ofthe inference accuracy, the inference precision, and the inferencerecall of the machine learning model after training or retraining themachine learning model on training data including the regulated streamof input data. The node metrics engine of a network node, such as thenode metrics engine 213, is configured to generate these metrics for theoutput of a model deployed on the network node.

The node metrics engine 213 can be configured to receive or generateground-truth labels after the model 217 generates and sends a modeloutput to the data source 215. The data source 215 can be configured toobtain confirmation as to the accuracy of a model output, for examplefrom user input or through independent and automatic mechanisms forverifying the model output. In response, the data source 215 can providefeedback to the model output, which the node metrics engine 213 can useto generate metrics as described herein.

The CMP 228 may also receive predetermined thresholds for a minimumoutput quality of a model deployed on the network node 210. For example,the CMP 228 may receive a minimum output quality threshold specifying99% recall for the deployed model on received input data. In otherexamples, the CMP 228 may provide its own minimum threshold, if one isnot provided.

The CMP 228 can be implemented as a machine learning model trained togenerate data stream policies, as described herein. The CMP 228 canreceive, as training data, one or more training examples of the outputquality of models deployed on various network nodes, labeled withcharacteristics of a data stream provided by the network node to theCMP. Training examples can include data characterizing one or more ofthe inference accuracy, the inference precision, and the inferencerecall of a deployed machine learning model. The labels for the trainingexamples can include data characterizing one or more of the rate of thestream of input data, the volume of the stream of input data, and thetypes of data in the stream of input data transmitted by the networknode to the CMP. The CMP 228 can be trained according to a variety ofapproaches, for example as a supervised machine learning model trainedusing stochastic, batch, or mini-batch gradient descent. The trainingdata can be generated by the ingestion engine 208, configured to receiveboth regulated data streams and metric data 237 from the node metricsengine 213.

In addition or alternatively, the training examples include additionaldata further characterizing a network node with a deployed machinelearning model. The node metrics engine of a network node, for example,the node metrics engine 213 of the network node 210, can be configuredto collect values for different metrics at least partiallycharacterizing the network node itself, including received data from thedata source(s), and/or the data source(s) themselves.

For example, the node metrics engine 213 can collect data related to thedeployment of the network node itself. This data can include physicalcharacteristics of the network or of the network node itself, e.g., thelocation of one or more computing devices or processors implementing thenetwork node, or the type of hardware or infrastructure the network nodeis built using. The data at least partially characterizing the networknode can include characteristics of one or more streams of data receivedby the network node, such as the rate, volume, and types of datatransmitted to the network node from one or more data sources. Thesecharacteristics can also include temporal information, such as how oftendata is received, and at what rates data is received in a stream by thenetwork node over different periods of time.

The ingestion engine 208 can receive data from network nodes and/orother devices in communication with the system 100 over the network 120.The ingestion engine 208 can distribute data to other components of thesystem 100, such as the CMP 228 and the training engine 204. The datareceived can include regulated data streams A, B 222A-B, as well as datastreams that are unregulated, which may be received by a network nodefor which the CMP 228 has not generated a policy for a correspondingdata stream.

Data streams received can be labeled by the ingestion engine 208 withidentifiers, for example based on the origin of the data stream. Anexample identifier can be a tuple, for example in the form: <nodeidentifier, data source identifier, data stream type>, specifying thenetwork node from which the stream is received, the data source fromwhich the network node received the stream, and the type of data, suchas raw bytes or encoded data, respectively.

The ingestion engine 208 can also receive the metric data 237 generatedby the node metrics engine 213. The regulated data streams A, B 222A-Bcan be sent to the training engine 204, which can be configured fortraining models that are deployed on various network nodes. The metricdata 237 can be labeled with characteristics of input data streamsreceived by the model 217, and be sent to the CMP 228 for training, asdescribed herein.

The training engine 204 can train machine learning models according toany of a variety of training procedures, including supervised,unsupervised, and semi-supervised training approaches. Before themachine learning models are fully trained, tested, and deployed onrespective network nodes, the training engine 204 can generate trainingdata from the data streams 216A, B, labeled according to providedground-truth labels. Similar to the labels of the training data for theCMP 228, the labels provided to the training engine 204 can be providedas feedback to the node output 232 provided to the data source 215.

The deployment engine 206 can be configured to send a trained model 227trained by the training engine 204 to a corresponding network node. Thedeployment engine 206 can maintain data associating various networknodes with corresponding machine learning models. For example, eachnetwork node can execute one or more machine learning models as part ofits respective inferencing engine, described herein. The one or moremachine learning models executed by one network node, such as thenetwork node 210, can at least partially overlap with one or moremachine learning models of another network node. In some examples, theone or more machine learning models of one network node can becompletely different than machine learning models implemented by anothernetwork node.

Telecommunication orchestrators and controllers are often deployed tomanage network element configurations. Sometimes those configurationchanges may lead to network anomalies. Upon detecting such anomalies,the CMP 228 would recommend rollback of configuration changes to theorchestrators or controllers. Such recommendations may be specific tosome deployments, or applicable systems-wide.

Example Methods

FIG. 3 is a flowchart of an example process 300 for regulating a streamof output data for a network node of a distributed network. For example,the data management system 100 as described herein with reference toFIGS. 1-2 can perform the process 300.

The system receives node metrics data at least partially characterizinga network node executing a deployed machine learning model on a streamof input data, according to block 310. As described herein withreference to FIGS. 1-2 , the node metrics data can at least partiallycharacterize a network node, which may include characterizing differentstreams of input data to the network node. The process 300 is describedherein with reference to a single data stream to a single network node,although it is understood that in other examples, multiple network nodescan be in communication with the data management system performingmultiple instances of the process 300, in parallel or in sequence.

The system generates a policy for regulating the stream of input datatransmitted by the network node, according to block 320. The stream ofinput data can be received by the network node from a data source, asdescribed herein with reference to FIGS. 1-2 . As described herein withreference to FIG. 2 , the system can generate a respective per-datastream policy for each data stream. The policy can specify how a networknode is to control the rate, volume, and types of data, among otherthings, to be transmitted to a central management plane for the system.The system can train a CMP to generate policies using training data ofdifferent quality metrics for different models deployed across networknodes of a distributed system, labeled with data characterizing one ormore of the rate of the stream of input data, the volume of the streamof input data, and the types of data in the stream of input datatransmitted by the network node to the CMP, among other quantifiablecharacteristics of the stream of input data.

The system sends the policy to the network node, according to block 330.The network node can be configured to execute the policy, for example bytransmitting a regulated stream of input data to the CMP of the systemwith characteristics matching or approximating characteristics specifiedin the provided policy. As an example, if the policy specifiedtransmitting data only during certain time periods, the network node canbe configured to execute the policy by causing data to be transmittedonly during those certain time periods.

FIG. 4 is a flowchart of an example process 400 for initializing a newnetwork node of the distributed network, according to aspects of thedisclosure. Initialization can refer to a process in which a computingplatform connects to a new network node, for example to communicate dataand to train a machine learning model for deployment on the networknode. The network node can be created by allocating computing resourcesof the platform, or in some examples, created as one or more computingdevices previously not connected to the platform. In those examples,initialization can include the process by which the platform connects tothe new network node and begins communication.

As with the network nodes described herein with reference to FIGS. 1-2 ,the new network node presently described can also receive one or moredata streams from one or more different data sources. Initially, thenetwork node can send one or more streams of input data to the CMP fortraining, and to receive a trained machine learning model to deploy onthe network node. The system can perform the process 300 describedherein to generate policies for each data stream received by the newnetwork node. In some examples, the data management system can performthe process 400, as part of initializing a new network node.

The CMP receives metric data from an initialized network node, accordingto block 410. An initialized network node can be a network node with adeployed machine learning model trained by the data management system.The initialized network node can be configured to send streams of inputdata to the CMP, but may not do so according to a policy as describedherein.

The CMP determines whether the metric data of the initialized networknode is similar to metric data of a second network node of thedistributed network within a similarity threshold, according to diamond420. The similarity threshold can be predetermined, for example based onempirical or statistical analysis of different data streams anddifferent metrics having statistically significant correlations betweenthe data stream and the policy applied to the stream. The similaritythreshold can be multi-dimensional, meaning that several metrics atleast partially characterizing the network node can be compared betweenthe initialized network node and other network nodes of a distributednetwork.

If the data management system determines that there is no metric datasimilar to the metric data of the initialized network node (“NO”), thenthe process 400 ends. If the CMP determines that the metric data of thesecond network node is similar to the metric data of the initializednetwork node within a similarity threshold (“YES”), then the CMP sendsthe policy corresponding to the second network node, to the initializednetwork node, according to block 430. The second network node can be anynetwork node for which the system has generated at least one policy.

The sent policy can bootstrap the regulation of data at the initializednetwork node, before the CMP generates a tailored policy based on nodemetrics data received from the new network node. In this way, the systemcan manage newly executed nodes to begin to balance output quality andoperational cost. At scale, an approximated policy based on similaritiespreviously generated policies can quickly reduce operational costs whenmany network nodes are initialized, over not providing any form ofregulation at all, or providing a uniform policy which may not beindividually suited for a deployed node.

The data management system updates the policy of the initialized networknode based on received metric data, according to block 440. For example,the data management system can generate a policy for a data streamreceived from the initialized network node, according to the process 300as described herein with reference to FIG. 3 . In some examples, thedata management system may not update the policy of a data stream of theinitialized network node. In those examples, one reason for not updatingthe policy is because the policy provided performs better according tothe applied objectives, such as model output quality and operationalcost, than any other policy generated by the data management system.

The time at which the data management system updates the policy can beany length of time after sending the initial policy to the network node,as described according to block 430. For example, the data managementsystem can send an updated policy to the network node as soon as one hasbeen generated, or the data management system can generate the updatedpolicy at a predetermined subsequent time, for example after apredetermined amount of time has passed to allow for additional inputdata to be received by the data management system. The predeterminedamount of time can be configured, for example based on user input, or atthe time the data management system is implemented on the platform.

FIG. 5 is a flowchart of an example process 500 for regulating a streamof output data on a network node of a distributed network, according toaspects of the disclosure.

A network node sends node metrics data to a data management system,according to block 510. As described herein with reference to FIGS. 1-2, the network node can implement a node metrics engine configured togenerate metrics at least partially characterizing the network node anddata streams received by the network node from one or more data sources.

The network node receives a policy generated by a data managementsystem, according to block 520. The data management system can generatethe policy using the node metrics data sent to the system by the networknode, according to block 510.

The network node regulates the stream of data according to the receivedpolicy, according to block 530. The network node can be configured toconvert the received policy into one or more instructions executable bythe network node to cause the network node to adjust characteristics ofa stream of input data transmitted to the data management system, inaccordance with the policy.

Example Computing Environment

FIG. 6 is a block diagram of an example computing environment 600 forimplementing the data management system 100. The system 100 can beimplemented on multiple devices having one or more processors in one ormore locations, such as one or more server computing devices 615 of acomputing platform 601. The system 100 can communicate with multiplenetwork nodes, such as network node 612 and network node 645. Forexample, the server computing device(s) 615 can make up at least part ofthe computing platform 101 of FIG. 1 , and implement the centralmanagement plane 102, as well as other components, such as the trainingengine 204, deployment engine 206, and the ingestion engine 208 of thesystem 100. As another example, the network node 612 can implement aninferencing engine 699 and a data source regulator 698.

Network node 612 and the server computing device(s) 615 can becommunicatively coupled to one or more storage devices 630 over anetwork 660. The storage device(s) 630 can be a combination of volatileand non-volatile memory, and can be at the same or different physicallocations than the computing devices 612, 615. For example, the storagedevice(s) 630 can include any type of non-transitory computer readablemedium capable of storing information, such as a hard-drive, solid statedrive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM,write-capable, and read-only memories.

The server computing device(s) 615 can include one or more processors613 and memory 614. The memory 614 can store information accessible bythe processor(s) 613, including instructions 621 that can be executed bythe processor(s) 613. The memory 614 can also include data 623 that canbe retrieved, manipulated or stored by the processor(s) 613. The memory614 can be a type of non-transitory computer readable medium capable ofstoring information accessible by the processor(s) 613, such as volatileand non-volatile memory. The processor(s) 613 can include one or morecentral processing units (CPUs), graphic processing units (GPUs),field-programmable gate arrays (FPGAs), and/or application-specificintegrated circuits (ASICs), such as tensor processing units (TPUs).

The instructions 621 can include one or more instructions that whenexecuted by the processor(s) 613, causes the one or more processors toperform actions defined by the instructions. The instructions 621 can bestored in object code format for direct processing by the processor(s)613, or in other formats including interpretable scripts or collectionsof independent source code modules that are interpreted on demand orcompiled in advance. The instructions 621 can include instructions forimplementing components of the system 100 consistent with aspects ofthis disclosure. The system 100 can be executed using the processor(s)613, and/or using other processors remotely located from the servercomputing device(s) 615, such as the one or more processors 616 of thenetwork node 612.

The data 623 can be retrieved, stored, or modified by the processor(s)613 in accordance with the instructions 621. The data 623 can be storedin computer registers, in a relational or non-relational database as atable having a plurality of different fields and records, or as JSON,YAML, proto, or XML documents. The data 623 can also be formatted in acomputer-readable format such as, but not limited to, binary values,ASCII or Unicode. Moreover, the data 623 can include informationsufficient to identify relevant information, such as numbers,descriptive text, proprietary codes, pointers, references to data storedin other memories, including other network locations, or informationthat is used by a function to calculate relevant data.

The network node 612 can also be configured similar to the servercomputing device(s) 615, with one or more processors 616, memory 617,instructions 618, and data 619. In some examples, the network node 612can be a user computing device, such as a personal computer, asmartphone, a wearable device, or any other computing device configuredfor receiving user input and/or generating user output.

For example, the network node 612 can also include a user output 626,and an input 624. The user input 624 can include any appropriatemechanism or technique for receiving input from a user, such askeyboard, mouse, mechanical actuators, soft actuators, touchscreens,microphones, and sensors. The server computing device(s) 615 can beconfigured to transmit data to the user computing device(s) 612, and thenetwork node 512 can be configured to display at least a portion of thereceived data on a display implemented as part of the user output 626.The user output 626 can also be used for displaying an interface betweenthe network node 612 and the server computing device(s) 615. The useroutput 626 can alternatively or additionally include one or morespeakers, transducers or other audio outputs, a haptic interface orother tactile feedback that provides non-visual and non-audibleinformation to the platform user of the user computing device 612.

Although FIG. 6 illustrates the processors 613, 616 and the memories614, 617 as being within the server computing device(s) 615 and networknode 612, components described in this specification, including theprocessors 613, 616 and the memories 614, 617 can include multipleprocessors and memories that can operate in different physical locationsand not within the same computing device. For example, some of theinstructions 621, 618 and the data 623, 619 can be stored on a removableSD card and others within a read-only computer chip. Some or all of theinstructions and data can be stored in a location physically remotefrom, yet still accessible by, the processors 613, 616. Similarly, theprocessors 613, 616 can include a collection of processors that canperform concurrent and/or sequential operation. The server computingdevice(s) 615 and network node 612 can each include one or more internalclocks providing timing information, which can be used for timemeasurement for operations and programs run by the server computingdevice(s) 615 and network node 612.

The network node 612 and/or the server computing device(s) 615 can beconfigured to receive requests to process data or other input from usercomputing device 650. The computing device 650 can be another networknode connected to the distributed network 620, or a computing devicethat communicates with one or more nodes and/or the one or more servercomputing devices of the network 620.

The computing platform 601 in which the data management system 100 isimplemented can be configured to provide a variety of services to users,through various user interfaces and/or APIs exposing the platformservices. One or more services can be a machine learning framework or aset of tools for generating neural networks or other machine learningmodels according to a specified task and training data. The datamanagement system 100 can be configured to train and deploy one or moremachine learning models onto the multiple nodes of the network 620, asdescribed herein. The user computing device 650 may transmit and receivedata to and from the system 100, for example sending queries forprocessing by a deployed model, and receiving a prediction from themodel in response.

Network node 645 can be similarly configured to network node 612, and befurther configured to communicate with the server computing device(s)615 directly through the network 620, or indirectly through one or moreother network nodes, e.g., the network node 612. The devices, includingthe server computing device(s) 615, network nodes, such as network node612 and network node 245, and the user computing device 250 can becapable of direct and indirect communication over the network 620. Thedevices of the network 620 can set up listening sockets that may acceptan initiating connection for sending and receiving information.

The network 620 itself can include various configurations and protocolsincluding the Internet, World Wide Web, intranets, virtual privatenetworks, wide area networks, body area networks, personal areanetworks, near-me area networks, local area networks, campus areanetworks, telecommunication networks, including cellular networks,telephone networks, sensor networks, radio access networks (RAN), andbackbone or core networks (CN), private networks using communicationprotocols proprietary to one or more companies, etc. The network 620 canspan over different physical network infrastructures, maintained by oneor more different providers. The network 620 can implement any of avariety of distributed computing architectures or paradigms, such asclient-server based architectures, three-tier or multi-tierarchitectures, peer-to-peer architectures, distributed real-timesystems, distributed database systems, systems based on parallelprocessing techniques, decentralized networks, mesh networks, etc.

The network 620 can support a variety of short- and long-rangeconnections. The short- and long-range connections may be made overdifferent bandwidths, such as 2.402 GHz to 2.480 GHz, 2.4 GHz and 5 GHz;13.56 MHz; or with a variety of communication standards, such ascommunication standards for wireless broadband communication. Thenetwork 620, in addition or alternatively, can also support wiredconnections between the devices of the network 620, including overvarious types of Ethernet connection.

It is understood that the aspects of the disclosure can be implementedaccording to a variety of different configurations and quantities ofcomputing devices, including in paradigms for sequential or parallelprocessing, or over a distributed network of multiple devices. In someimplementations, aspects of the disclosure can be performed on a singledevice, and any combination thereof.

Aspects of this disclosure can be implemented in digital circuits,computer-readable storage media, as one or more computer programs, or acombination of one or more of the foregoing. The computer-readablestorage media can be non-transitory, e.g., as one or more instructionsexecutable by a cloud computing platform and stored on a tangiblestorage device.

In this specification the phrase “configured to” is used in differentcontexts related to computer systems, hardware, or part of a computerprogram, engine, or module. When a system is said to be configured toperform one or more operations, this means that the system hasappropriate software, firmware, and/or hardware installed on the systemthat, when in operation, causes the system to perform the one or moreoperations. When some hardware is said to be configured to perform oneor more operations, this means that the hardware includes one or morecircuits that, when in operation, receive input and generate outputaccording to the input and corresponding to the one or more operations.When a computer program, engine, or module is said to be configured toperform one or more operations, this means that the computer programincludes one or more program instructions, that when executed by one ormore computers, causes the one or more computers to perform the one ormore operations.

While operations shown in the drawings and recited in the claims areshown in a particular order, it is understood that the operations can beperformed in different orders than shown, and that some operations canbe omitted, performed more than once, and/or be performed in parallelwith other operations. Further, the separation of different systemcomponents configured for performing different operations should not beunderstood as requiring the components to be separated. The components,modules, programs, and engines described can be integrated together as asingle system, or be part of multiple systems.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the examples should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible implementations. Further, the same referencenumbers in different drawings can identify the same or similar elements.

1. A system comprising: a distributed network of a plurality of networknodes; and one or more processors in the distributed network, the one ormore processors implementing a central management plane (CMP), the CMPconfigured to: receive metric data at least partially characterizing afirst network node of the plurality of network nodes executing adeployed machine learning model on a stream of input data received bythe network node from a data source; generate a policy for regulatingthe stream of input data transmitted by the network node to the one ormore processors; and send the policy to the network node.
 2. The systemof claim 1, wherein the policy comprises one or more rules for adjustingone or more of: the rate of the stream of input data transmitted by thefirst network node to the CMP, the volume of the stream of input datatransmitted by the first network node, a time at which the stream ofinput data is transmitted by the first network node, and the types ofdata in the stream of input data transmitted by the first network nodeto the CMP.
 3. The system of claim 2, wherein the CMP is furtherconfigured to receive a regulated stream of input data, wherein theregulated stream of input data is the stream of input data after thepolicy is applied to the stream of input data; and wherein in generatingthe policy, the CMP is further configured to generate the policyaccording to a plurality of objectives based at least on: reducing theoperational cost of transmitting the regulated stream of input data overthe distributed network or using the regulated stream of input data totrain the deployed machine learning model, and increasing the outputquality of the deployed machine learning model according to one or morequality metrics, after training the model on the regulated stream ofinput data.
 4. The system of claim 3, wherein the operational cost oftransporting output data is further based on one or more of: the rate atwhich the regulated stream of input data is received by the CMP over aperiod of time, the volume of the regulated stream of input datareceived by the CMP, and the type of data in the regulated stream ofinput data.
 5. The system of claim 3, wherein the one or more qualitymetrics of the machine learning model measure one or more of theinference accuracy, the inference precision, and the inference recall ofthe machine learning model after training or retraining the machinelearning model on training data comprising the regulated stream of inputdata.
 6. The system of claim 5, wherein the machine learning model is afirst machine learning model, and wherein in generating the policy, theCMP is configured to process the metric data through a second machinelearning model trained to generate policies, the second machine learningmodel trained using labeled metric data: comprising featurescorresponding to one or more of the inference accuracy, the inferenceprecision, and the inference recall of the deployed machine learningmodel, and labeled with data characterizing one or more of the rate ofthe stream of input data, the volume of the stream of input data, andthe types of data in the stream of input data transmitted by the networknode to the CMP.
 7. The system of claim 3, wherein the plurality ofobjectives further comprises one or more objectives for training themachine learning model.
 8. The system of claim 1, wherein the stream ofinput data is a first stream of input data, the policy is a firstpolicy, and the metric data is first metric data; and wherein the CMP isfurther configured to: receive second metric data at least partiallycharacterizing the performance of the first network node executing thedeployed machine learning model on a second stream of input datadifferent than the first stream of input data, and generate and send asecond policy for regulating the second stream of input data, the secondpolicy different from the first policy.
 9. The system of claim 8,wherein the second stream and the first stream of input data arereceived from the network node by the same data source.
 10. The systemof claim 1, wherein the CMP is further configured to: receive anindication of a second network node added to the distributed network;receive metric data from the second network node; determine, based onthe metric data of the first network node and the metric data of thesecond network node, that the first and second network nodes are similarwithin a predetermined threshold; and in response to the determination,send the policy to the second network node.
 11. The system of claim 1,wherein the metric data is first metric data, the policy is a firstpolicy, and wherein the CMP is further configured to: receive secondmetric data from a second network node of the plurality of networknodes; generate a second policy using the first metric data, the secondmetric data, or both the first metric data and the second metric data,wherein the second policy is different than the first policy; and sendthe second policy to the second network node.
 12. The system of claim 1,wherein in regulating the stream of input data, the CMP is furtherconfigured to apply one or more rules of a policy to the stream of inputdata to adjust the transmission of data in the stream of input data. 13.The system of claim 1, wherein the metric data at least partiallycharacterizing a first network node of the plurality of network nodescomprises one or more of: physical characteristics of the distributednetwork or of the first network node, characteristics of one or morestreams of data received by the network node, and characteristics of thedata source.
 14. A computer-implemented method, comprising: receiving,by one or more processors, metric data at least partially characterizinga first network node of a plurality of network nodes executing adeployed machine learning model on a stream of input data received bythe network node from a data source, the plurality of network nodes partof a distributed network; generating, by the one or more processors, apolicy for regulating the stream of input data transmitted by thenetwork node to the one or more processors; and sending, by the one ormore processors, the policy to the first network node.
 15. The method ofclaim 14, wherein the policy comprises one or more rules for adjustingone or more of: the rate of the stream of input data transmitted by thefirst network node to the one or more processors, the volume of thestream of input data transmitted by the first network node, a time atwhich the stream of input data is transmitted by the first network node,and the types of data in the stream of input data transmitted by thefirst network node to the one or more processors.
 16. The method ofclaim 15, further comprising: receiving, by the one or more processors,a regulated stream of input data, wherein the regulated stream of inputdata is the stream of input data after the policy is applied to thestream of input data; and wherein generating the policy comprisesgenerating the policy according to a plurality of objectives based atleast on: reducing the operational cost of transmitting the regulatedstream of input data over the distributed network or using the regulatedstream of input data to train the deployed machine learning model, andincreasing the output quality of the deployed machine learning modelaccording to one or more quality metrics, after training the model onthe regulated stream of input data.
 17. The method of claim 16, whereinthe operational cost of transporting output data is further based on oneor more of: the rate at which the regulated stream of input data isreceived by the one or more processors over a period of time, the volumeof the regulated stream of input data received by the one or moreprocessors, and the type of data in the regulated stream of input data.18. The method of claim 16, wherein the one or more quality metrics ofthe machine learning model measure one or more of the inferenceaccuracy, the inference precision, and the inference recall of themachine learning model after training or retraining the machine learningmodel on training data comprising the regulated stream of input data.19. The method of claim 18, wherein the machine learning model is afirst machine learning model, and wherein generating the policycomprises, processing, by the one or more processors, the metric datathrough a second machine learning model trained to generate policies,the second machine learning model trained using labeled metric data:comprising features corresponding to one or more of the inferenceaccuracy, the inference precision, and the inference recall of thedeployed machine learning model, and labeled with data characterizingone or more of the rate of the stream of input data, the volume of thestream of input data, and the types of data in the stream of input datatransmitted by the network node to the one or more processors.
 20. Oneor more non-transitory computer-readable storage media encodinginstructions that when executed by one or more processors, cause the oneor more processors to perform operations comprising: receiving metricdata at least partially characterizing a first network node of aplurality of network nodes executing a deployed machine learning modelon a stream of input data received by the network node from a datasource, the plurality of network nodes part of a distributed network;generating a policy for regulating the stream of input data transmittedby the network node to the one or more processors; and sending thepolicy to the first network node.